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Introduction 

The  purpose  of  this  study  is  to  develop,  implement  and  evaluate  a  web-based,  self-paced 
training  program  to  provide  skill-improvement  oriented  continuing  education  for  mental  health 
professionals.  It  is  intended  to  reach  a  broad  range  of  VA  health  professionals  presenting 
content  that  is  highly  relevant  and  easy  to  access  for  practitioners.  It  follows  adult  learning 
principles  and  incorporates  interactivity,  practice,  and  modeling  of  effective  treatment  for 
Posttraumatic  Stress  Disorder  (PTSD).  Overall,  the  current  study  is  designed  to  demonstrate  in 
a  randomized,  controlled  design  that  web-based  CBT  training  is  a  cost-effective  means  for 
developing  relevant  clinical  skills  in  VHA  providers  responsible  for  the  clinical  care  of  veterans 
with  symptoms  of  PTSD.  The  specific  hypotheses  to  be  evaluated  are: 

1.  Web-based  training  will  increase  clinician  skill  in  core  CBT  techniques  compared  to 
usual  training  exposure  (control). 

2.  Web-based  training  followed  by  expert  group  consultation  will  increase  clinician  skill  in 
core  CBT  techniques  compared  to  web-based  training  alone  and  usual  training 
exposure  (control). 

3.  Web-based  training  will  increase  clinician  knowledge,  attitudes  and  self-efficacy  in 
evidence-based  counseling  practices  with  PTSD  patients. 

4.  Training  effects  will  be  mediated  by  participant  engagement  in  training,  as  measured 
by  modules  completed  and  expert  group  consultation  sessions  attended. 

Body 

This  section  of  the  report  shall  describe  the  research  accomplishments  associated  with  each 
task  outlined  in  the  approved  Statement  of  Work. 

1.  Prepare  Training  Content 

a.  Prepare  PTSD  and  revise  CBT  case  and  instructional  material  all  modules 

i.  Content  development  for  the  three  training  modules  (Motivational 

Interviewing,  Goal-Setting,  and  Behavioral  Task  Assignment)  was  completed 
during  the  first  year  of  the  project.  A  team  of  carefully-selected  expert  content 
developers  was  assembled,  comprised  of  the  project  investigators,  in  addition 
to  Amy  Naugle,  Ph.D.,  Jennifer  Sharpe  Potter,  Ph.D.,  J.  Gayle  Beck,  Ph.D., 
and  Brett  T.  Litz,  Ph.D.  These  latter  individuals  were  invited  to  serve  as 
special  consultants  to  the  project  based  on  their  extensive  experience  and 
knowledge  in  the  specific  content  areas. 

b.  Prepare  case  material  to  be  introduced 
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i.  During  the  first  year  of  this  project  we  developed  the  character  “Robert”,  a 
Vietnam  Era  Veteran  with  PTSD,  which  is  the  character  the  simulated  patients 
will  be  portraying.  Detailed  background  was  created  for  this  character. 

c.  Prepare  expert  group  supervisor  manual  and  materials 

i.  During  the  first  year  of  this  project  we  created  and  finalized  a  manual  and 
materials  for  the  simulated  patients. 

ii.  During  the  second  year  of  this  project  we  have  finalized  a  manual  and 
materials  for  the  consultant  procedures  for  the  consultation  sessions. 

d.  Training  content  complete,  reviewed  by  VHA  CBT  experts 

i.  During  the  first  year  of  this  project  we  finalized  the  training  content  that  will  be 
placed  on  the  web  for  training  purposes.  Furthermore,  the  training  modules 
were  reviewed  by  VHA  CBT  experts  at  the  National  Center  for  PTSD  in  Menlo 
Park,  CA  and  Boston,  MA,  as  well  as  by  qualified  study  staff. 

2.  Prepare  Research  Protocol 

a.  Eligibility  criteria,  exclusion  criteria,  screening  protocol  refined 

i.  During  the  first  year  of  the  study  we  finalized  the  eligibility  criteria  (must  be  a 
full-time  VHA  mental  health  clinician  that  works  primarily  with  PTSD  patients), 
and  exclusion  criteria  (does  not  deliver  treatment  in  their  daily  practice,  is  not 
a  VHA  mental  health  clinician,  and  those  that  have  participated  in  the  Beck"s 
CBT  training  for  depression,  as  this  training  is  closely  related  to  the  web 
training  program). 

b.  Sample  frame,  web  contact  methods,  email  lists  identified 

i.  During  the  first  year  of  the  study  we  identified  potential  participants  as  well  as 
created  several  email  lists  of  potential  participants 

1.  Email  lists  included:  VHA  mental  health  clinicians  that  have 
participated  in  the  “Clinical  Training  Program”  at  the  National  Center 
for  PTSD,  team  leaders  at  each  Vet  Center  in  the  United  States,  and 
the  National  Mentor  Program  in  PTSD. 

2.  Announcements  were  made  on  national  phone  calls  and  presentations 
to  appropriate  audience  members. 

3.  Postcards  were  created  and  disseminated  at  conferences  and  other 
mental  health  events.  See  Figure  1. 

c.  Simulated  patient  interview  specified  and  scripted 
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i.  In  Year  1  a  manual  was  created  for  the  simulated  patient  interview  process. 
This  manual  includes  detailed  characteristics  of  “Robert”,  a  precise  script  for 
instructions  during  the  simulated  patient  interview,  mandatory  statements  to 
be  said  during  interview  for  evaluation  purposes  by  the  simulated  patient,  a 
draft  of  decision  rules  for  the  simulated  patient  if  they  are  presented  certain 
situations  in  the  interview,  and  finally,  a  detailed  set  of  procedures  for  the 
simulated  patient  interview. 

d.  Research  measures  modified  to  reflect  PTSD  CBT  focus 

i.  The  training  modules  were  developed  in  Year  1  to  reflect  PTSD  and  CBT 
focus.  All  examples  on  the  training  modules  pertain  to  patients  with  PTSD. 

e.  Consent  form  drafted,  human  subjects  protocol  finalized 

i.  During  the  first  year  of  the  study  the  web-based  consent  form  was  created. 
The  terms  of  the  study  were  clearly  defined  with  an  accept"  participation 
button.  Those  who  chose  accept  moved  to  the  next  phase  of  the  study  (the 
consent  form  was  approved  by  both  the  Stanford  and  NERI  IRBs).  . 

f.  NERI/VHA  IRB  approvals  and  USAMRMC  HRPO  human  subject  protocol  approval 

i.  During  the  first  year  of  the  study  the  protocol  was  approved  by  the  Stanford 
and  NERI  IRBs,  VA  Research  and  Development,  and  finally  by  USAMRMC 
HRPO.  During  the  second  year  of  study  a  continuing  review  was  approved  by 
the  Stanford  and  NERI  IRBs  and  VA  Research  and  Development. 

g.  Pilot  test  all  study  procedures  and  materials  (prior  to  programming) 

i.  Repeated  simulations  of  the  simulated  patient  interview  were  performed  and 
evaluated  to  assure  that  procedures  were  consistently  executed. 

Furthermore,  repeated  beta  testing  of  the  web-based  materials  was 
conducted  by  the  study  team  in  Year  1  of  the  study. 

3.  Produce  Web  Site  and  Instructional  Program 

a.  VHA  web  host  programmers  provide  specification  and  guidance  to  web  programmers 
and  database  programmers. 

i.  We  have  obtained  VHA  web  host  programming  instructions  and  have 
implemented  them  in  the  construction  of  our  web-based  training  program. 
Currently,  the  web-based  training  program  will  not  be  on  a  VHA  server  as 
updates  are  currently  being  performed  for  this  server  which  will  not  allow  our 
program  to  be  uploaded  in  a  timely  manner  to  execute  the  study.  The  website 
is  hosted  on  the  server  where  it  is  being  developed  currently.  The  site  was 
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programmed  with  the  intention  and  ability  to  transport  the  site  to  another 
server  and  plans  are  to  discuss  the  ultimate  hosting  location  and  URL  during 
the  final  study  year. 

b.  PTSD  case  material  completed 

i.  During  the  first  year  of  this  project  we  developed  the  character  “Robert”,  a 
Vietnam  Era  Veteran  with  PTSD,  which  is  the  character  the  simulated  patients 
will  be  portraying.  Detailed  background  was  created  for  this  character. 

c.  Web  programming  specifications  completed 

i.  During  the  first  year  of  this  project  web-programming  specifications  were 
identified  and  completed. 

d.  Web  modules  are  programmed 

i.  During  the  first  year  of  this  project  programming  was  completed  to  host  the 
web  training  program  and  online  evaluation  materials. 

e.  Web  program  transferred  to  VHA  intranet  server 

i.  During  development  and  based  on  meetings  with  web  programmers  from  the 
Central  Office  it  was  determined  that  the  aspects  of  the  VHA  intranet  server 
were  under  development  which  could  impact  programming  and  hosting  of  the 
PTSD  for  CBT  site.  It  was  determined  that  the  best  course  of  action  was  to 
host  the  site  on  the  server  where  it  was  developed  during  the  study.  During 
the  final  year  of  the  study  and  after  data  collection  is  completed  we  will 
discuss  the  possibility  of  transferring  the  web  program  to  VHA  intranet  server. 

f.  Beta  test  and  modify  the  working  web  program 

i.  Both  the  web  training  modules,  study  screener,  consent,  and  the  evaluation 
materials  were  beta  tested  and  used  in  Year  2  of  the  study. 

4.  Develop  Study-specific  Data  Management  System 

a.  Flow  chart  all  study  steps,  web  data  collection  and  database  requirements 

i.  During  the  first  year  of  this  project  a  flow  chart  of  all  study  steps  was  created. 
Simultaneously  study  staff  evaluated  web  data  collection  and  database 
requirements  and  relayed  this  information  to  the  web  programmers.  See 
Figure  2  for  flow  chart. 

b.  Develop  web-based  data  forms  for  all  research  measures 

i.  During  the  first  year  of  the  project  we  have  developed  all  web-based  data 
forms  for  all  research  measures.  This  process  was  performed  parallel  to  the 
development  of  the  research  measures. 
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c.  Program  the  database  including  project  data  entry  forms 

i.  During  the  first  year  of  the  project  the  database  for  project  data  was 
successfully  programmed,  was  beta  tested,  and  implemented  in  Year  2. 

d.  Program  web  usage  statistics  reports 

i.  During  the  first  year  of  the  project  usage  statistics  reports  were  created  and 
programmed  by  study  staff.  These  reports  were  beta  tested  and  implemented 
in  Year  2. 

e.  Program  reports  to  monitor  project  operations  and  minimize  drop  out 

i.  Using  the  data  management  system  on  the  NERI  server  numerous  reports  to 
monitor  project  operations  and  recruitment  were  created.  These  reports  are 
updated  as  information  is  entered  into  the  site,  making  project  operations 
more  efficiently  monitored. 

f.  Program  automatic  email  reminders/interaction  with  web  participants 

i.  During  the  first  year  of  the  study  we  have  created  automated  emails  in 
conjunction  with  the  programming  of  the  usage  reports  and  the  monitoring 
system  for  project  operations.  Separate  emails  are  created  for  participants  in 
each  of  the  three  treatment  groups  of  this  study.  Furthermore,  automated 
email  templates  have  been  created  as  reminders  of  key  timepoints  during 
scheduling  of  simulated  patient  interviews  and  the  consultation  sessions. 

g.  Beta  test  all  programmed  pieces  and  interactivity  with  web  course 

i.  Betas  testing  of  all  programmed  pieces  were  performed  and  completed  in 
Year  2  to  assure  that  there  are  no  interactivity  errors  with  the  web  course  or 
the  evaluation  materials. 

h.  Ongoing  monitoring  of  study  process 

i.  Study  recruitment  began  in  Year  2  in  October  of  2009.  Study  recruitment  was 
completed  by  July  31  of  2010. 

5.  Prepare  Expert  Rating  Protocol 

a.  Modify  CBT  Rating  Guide  to  reflect  web  course  objectives  and  PTSD  core  skills 

i.  The  CBT  Rating  Guide  for  the  simulated  patient  interviews  were  completed  in 
Year  2. 

b.  CBT  advisors  comment  on  Rating  Guide  and  methods 

i.  CBT  advisors  have  been  identified  to  help  with  the  construction  and 
evaluation  of  the  Rating  Guide. 

c.  Train  independent  study  raters  until  100%  concordance 
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i.  Raters  have  been  recruited  and  trained  according  to  a  detailed  training 
manualBased  on  the  development  of  the  rating  system,  training  sessions 
conducted,  and  available  literature  we  have  a  70%  expectation  of 
concordance,  given  the  complexity  of  the  Rating  Guide  to  reflect  the  web 
course  objectives  and  PTSD  core  skills. 

d.  Perform  blinded  independent  ratings  (n  =  120  *  2  =  240)  and  re-ratings 

i.  During  the  recruitment  period  168  participants  were  recruited  for  this  study. 
We  have  recruited  more  than  the  required  120  participants  in  anticipation  of 
attrition  and  due  to  the  high  volume  of  interest  during  recruitment.  A  random 
sample  of  20%  of  the  simulated  patient  interviews  will  be  rated  by  multiple 
raters  to  access  reliability  of  the  rating  system.  Ratings  of  interviews  began  in 
July  of  2010  and  are  estimated  to  be  completed  by  January  of  201 1 . 

6.  Data  Collection,  Intervention,  and  Simulated  Patients  (SP)  Interviews 

a.  Web  screening  of  potential  clinician  applicants,  estimated  screenings  200 

i.  During  the  first  year  of  the  project  a  web  screening  tool  was  created.  During 
Y02,  this  screener  was  implemented. 

b.  Schedule  and  carry  out  SP  baseline  interviews  with  eligible  clinicians,  n  =  120 

i.  Recruitment  for  this  study  began  in  October  of  2009.  We  have  exceeded  the 
goal  of  120  participants  and  have  closed  recruitment  with  168  participants 
enrolled. 

c.  Random  assignment  of  participant  with  completed  SP  interview  by  statistician  to  web- 
course,  web-course+supervision,  or  control  (n  =  120) 

i.  Random  assignment  of  participant  with  completed  simulated  patient  interview 
has  been  programmed  to  randomly  assign  to  treatment  groups  based  on 
experience  to  assure  that  clinical  experience  is  balanced  among  all  three 
treatment  groups.  Recruitment  was  completed  in  Year  2. 

d.  Participants  complete  assigned  condition 

i.  Web  course  participants  complete  instructions  (6  weeks  scheduled) 

1 .  Recruitment  began  and  was  concluded  in  Year  2. 

ii.  Web  course+supervision  completed  (12  weeks  scheduled) 

1 .  Recruitment  began  and  was  concluded  in  Year  2. 

iii.  Control  condition  completed  (12  weeks  scheduled) 

1 .  Recruitment  began  and  was  concluded  in  Year  2. 

e.  Schedule  and  carry  out  post-training  SP  Interviews  (Goal  is  80%  follow-up  rate) 
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i.  Recruitment  for  this  study  concluded  on  July  31  of  2010.  At  this  report,  not  all 
participants  have  had  the  opportunity  to  complete  their  post-training  SP 
Interviews  based  on  the  study  timeline. 

f.  Clinicians  in  training  conditions  submit  Counseling  session  Forms  (CSF)  and  receive 
individualized  graphic  feedback  (n  =  120  *  3+  =  360+) 

i.  A  “CBT  Session  Checklist”  has  been  created  for  participants  in  the 

consultation  treatment  group  to  monitor  the  use  of  learned  skills  in  current 
practice.  All  groups  will  be  asked  about  skill  usage  during  the  midpoint  and 
final  study  evaluations. 

7.  Analyses  and  Evaluation 

a.  Monitor  data  collection  rates  and  data  quality 

i.  Programming  has  been  completed  to  monitor  data  collection  rates  and  data 
quality.  Furthermore,  a  graphical  report  was  created  at  the  end  of  each 
month  during  recruitment  to  monitor  recruitment  progress  compared  to  the 
rate  we  will  need  to  recruit  at  to  complete  recruitment  by  the  end  of  the  year 
2010. 

b.  Create  interim  and  final  analytic  data  sets  and  basic  frequencies  for  Pl"sand  study 
investigators 

i.  Data  collection  remains  in  process  at  the  end  of  Year  2.  Interim  data  sets  and 
basic  frequencies  for  Pl"sand  study  investigators  are  currently  available. 

c.  Author  and  co-author  evaluation  findings,  project  reports  to  funding  source,  VPIA 
administrators  and  other  stake-holders,  scientific  papers  for  the  PTSD  mental  health 
community 

i.  Data  collection  is  in  process  as  expected  at  the  end  of  this  reporting  period. 
Interim  data  sets  and  basic  frequencies  for  PT'sand  study  investigators  are 
currently  available.  The  Statistical  Analysis  Plan  is  under  development,  and 
data  cleaning  activities  are  planned  to  begin  after  data  collection  is  completed 
in  October  201 1 .  Manuscript  planning  is  underway  and  methodological 
papers  are  currently  under  preparation.  The  main  results  paper  is  under 
discussion  and  will  be  the  initial  data  paper  produced  on  the  study. 

d.  Host  periodic  discussion,  solicit  written  comments  on  evaluation  findings  to  identify 
strengths  and  potential  barriers  for  bringing  program  to  scale,  within  and  outside  the 
VHA 
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i.  Project  PI  disseminated  the  study  procedures  and  design  at  the  Military 
Health  Conference  in  Kansas  City,  August  of  2009. 

ii.  Poster  presentation  on  Simulated  Patient  Interview  development  and  process 
at  World  Congress  of  Behavioral  and  Cognitive  Therapies  in  Boston,  June  of 
2010. 

Key  Research  Accomplishments 

•  Prepared  training  content 

•  Prepared  research  protocol 

•  Produced  web  site  and  instructional  program 

•  Developed  study-specific  data  management  system 

•  Prepared  expert  rating  protocol 

Reportable  Outcomes 

•  Project  PI  disseminated  the  study  procedures  and  design  at  the  Military  Health 
Conference  in  Kansas  City,  August  of  2009. 

•  Poster  presentation  on  Simulated  Patient  Interview  development  and  process  at  World 
Congress  of  Behavioral  and  Cognitive  Therapies  in  Boston,  June  of  201 0. 

Conclusion 

Data  collection  has  not  been  completed  at  this  time,  study  conclusions  are  not  available  at  this 
time.  The  work  planned  for  year  3  of  the  study  is  summarized  as  follows: 

1)  Complete  Data  Collection:  Enrollment  was  completed  in  July,  2010.  All  active  study 
participants  will  continue  through  the  study  protocol.  The  last  participants  enrolled  are 
estimated  to  complete  data  collection  October  31,  2010. 

2)  Complete  Rating,  Reliability  Re-Rating,  and  Quality  Control  (double  data  entry):  The 
assessment  form  for  the  main  study  outcome  was  completed  during  the  second  year  of  the 
project.  Raters  were  trained  in  July,  2010,  and  a  process  for  reliability  was  developed. 

Rating  has  begun  using  the  assessment  form  and  all  forms  will  be  data-entered  into  the  data 
management  system  upon  completion  of  programming.  All  data  entry  will  occur  in  the  third 
year  of  the  project. 

3)  Data  Cleaning:  Data  cleaning  will  occur  in  the  third  year  of  the  project  when  data  collection 
and  data  entry  have  been  completed.  The  data  management  system  has  a  procedure  in 
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place  for  double  data  entry  of  forms  to  protect  against  data  entry  errors.  Missing  data  will  be 
identified  and  confirmed  and  any  other  outstanding  data  issues  will  be  resolved. 

4)  Analysis:  The  statistical  analysis  plan  is  under  discussion  and  will  be  developed  during  the 
first  quarter  of  Y03.  Once  completed  data  cleaning  and  analysis  for  the  main  results  paper 
will  begin  immediately. 

5)  IRB  Annual  Review:  In  the  third  year  the  study  will  submit  an  Annual  Continuing  Review 
Report  to  the  NERI  and  Stanford  IRBs. 

6)  Manuscript  Development:  Plans  for  manuscripts  have  already  begun  and  will  be  continued  in 
the  third  year.  There  are  immediate  plans  to  develop  several  manuscripts  about  study 
design  and  processes  that  do  not  require  data.  A  manuscript  will  be  prepared  using  the 
main  study  outcomes  upon  completion  of  analyses  of  these  data.  Manuscripts  based  on 
additional  data  and  secondary  outcomes  will  be  considered.  Current  manuscripts  are 
underway  on  three  topics:  main  results,  simulated  patient  interview  methods,  and  telephone 
consultation  methods. 
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Figure  1:  Recruitment  Postcards 


Online  Training  in  CBT  Skills  for  PTSD  Treatment: 

Research  Project  Launching  in  September  2009 


Visit  our  website  to  sign  up  and  we  will  contact  you  when  enrollment  begins. 


Useful  Training! 

•CEUs  •  Self-paced  -  Interactive 
•Focused  on  3  key  components  of  CBT 


To  be  eligible  you  must  be: 
A  Mental  Health  Clinician 


-  A  Full  time  VHA  or  Vet  Center  employee 
A  Clinician  who  treats  veterans  with  PTSD 
Able  to  use  the  web  at  work  or  at  home 
■  Willing  to  consent  to  the  research  study 


-  Your  employment  will  not  be  affected  in  any  way 
regardless  of  whether  or  not  you  participate. 

Your  individual  information  will  be  kept  strictly  confidential. 

For  more  information  contact:  Project  PI: 

Donn  Garvert  Josef  Ruzek 

Clinical  Project  Coordinator  Director.  Dissemination  and  Training  Division 
National  Center  for  PTSD  National  Center  for  PTSD 

VA  Palo  Alto  Health  Care  System  VA  Palo  Alto  Health  Care  System 
donn. garvert '3va.gov 

For  general  information  about  participants  rights,  contact  866-680-2960. 

This  research  project  is  Ijrdfd  by  :he  Department  of  Defense,  US  Army  Medical  Research,  and  Materia!  Command, 
Congressional!*1  Directed  Uedical  Research  Program  ICOUSP)  as  a  PTSD  Investigator  Initiated  Research  Award. 
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Figure  2:  Diagrammatic  Representation  of  Evaluation  Design. 
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